Automated patch assessment for program repair at scale

Abstract In this paper, we do automatic correctness assessment for patches generated by program repair systems. We consider the human-written patch as ground truth oracle and randomly generate tests based on it, a technique proposed Shamshiri et al., called Random testing with Ground Truth (RGT) in paper. build curated dataset of 638 Defects4J 14 state-of-the-art systems, evaluate automated dataset. The results study are novel significant: First, improve state art performance RGT 190% improving oracle; Second, show that is reliable enough to help scientists overfitting analysis when they systems; Third, external validity knowledge largest ever.